In present applications, the volume of data that is available and used in the organizations is rapidly increasing. The data may be used for various roles such as decision making, monitoring of various tasks, analysis of financial data, etc. Typically, those organizations that effectively and efficiently manage large volumes of data, and use the information to make business decisions, will realize a significant competitive advantage in the marketplace.
In order to manage high volumes of data in terms of improvements in price, performance, scalability, and robustness of open computing systems, support systems which are sometimes referred to as On-Line Analytical Processing (OLAP) systems have been developed. OLAP systems allow data analysts to intuitively, quickly, and flexibly manipulate operational data using familiar business terms, in order to provide analytical insight into a particular problem or line of inquiry.
OLAP applications span a variety of organizational functions. For example, OLAP systems are used in finance departments, sales analysis, marketing etc. Finance departments use OLAP for applications such as budgeting, activity-based costing (allocations), financial performance analysis, and financial modeling. Sales analysis and forecasting are two of the OLAP applications found in sales departments. Among other applications, marketing departments use OLAP for market research analysis, sales forecasting, promotions analysis, customer analysis, and market/customer segmentation. Typical manufacturing OLAP applications include production planning and defect analysis.
High volume of data is gathered in a system through multiple channels. For example, in a retail system, customer's transactional data may be collected from point of sale, ecommerce and mobile transactions. Similarly customer's behavior data may be collected via customer's social media account and customer's feedback can come from multiple online and offline channels. All these numerously different fields of data is collected and stored by an Online Transactional Processing System (OLTP) that may be potentially spread across different machines where each system may have a different representation of the data that is collected.
A requirement of a multi-tenant database system in Software as a Service (SAAS) model is to store data collected across multiple organizations in the same schematic representation under the same logical entity, like database tables. The data is identified back to their respective organizations with the help of an identifier which is also saved with the record.
An extract, transform and load (ETL) system is responsible for extracting information and/or data from a source, transforming the data and further saving it in a format conducive for further use. However, current ETL system generally do not provide a scalable framework for working in a multi-tenant federated setup with diverse source schematic representations that allows for pluggable instructions. Therefore, there is a need to provide an ETL system which allows for OLAP cubes formation in a federated and multi-tenant setup via pluggable instruction sets.